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ABSTRACT 

A major theme of this series; of papers is the need to 
build evaluation and experimentation as a component part of 
educational planning. This paper addresses the structure of the 
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training of evaluators. The training discussion includes the possible 
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intensive training courses, regional and national training courses, 
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of typical evaluation projects. (Author/IST) 
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Evaluation and experimentation as a oomponent part of educational plann lnAr 

One major these of the preceding papers has been the need to build 
in evaluation and experimentation as a component part of educational planning;. 
To recapitulate some of the examples of the types of questions where evaluation 
• and experimentation can shed light: 

- Are the revised or new curriculum materials \workinjg' with the 
students for whom they are intended, in the sense that the students 
are achieving^ the learning objectives? If not, what are the weak 
points in what is being attained by which students? 

- Is there a systematic increase over time of student achievement including?; 
practical subjects)? Is this equal for all sub-groups of sJtudd^its 

in different schools in different regions within the country? If 
inequalities exist, where do they exist? 

- Are the objectives of teacher education courses being attained by 
all students? If not, why not? What remedial measures can be taken? 

- Are the teacher behaviours taught In teacher education having a 
positive effect on cognitive learning, affective learning, etc., 
in the persons taught by those teachers? If not, are there other 
behaviours which should be taught? If so, which? 

The answers to questions such as the above are important for planners 
in any system of education and help in the reallocation of resources. They 
can improve understanding of the way in which the various factors (input and 
process into the educational system) interact with each other so that 
improvements can be gained through the manipulation of such variables. Different 
models of evaluation and experimentation have already been given in some of the pre- 
ceding papers- We recommend that donor agencies build evaluation and experimentation 
into its future projects to the extent possible for the good of the recipient 
coiintry (as well as the donor agency) and that the donor agency and the recipient be 
willing to allocate such resources in order that systematically designed experiment 
and evaluation can be undertaken. In this way all parties can learn from differing' 
educatiur^l practices. Indeed, if evaluation data exist in countries, the 
donor agencj's identification of the weak aspects in need of support will be an 
easier task. Furthermore, consiaering the large percentage of the GNP and 
National Budget which is spent on education in most developing countries, it 

5 



is almost Inconceivable that little systematic effort in evaluation of education 
has been undertaken. Given the * state of the art' in evaluation, it would be 
feasible to introduce evaluation at relatively small cost in moat developing 
countries. 

There are several examples of large-scale evaluation and experimentation 
In developed countries and some in developing countries. The tntemational 
Association for the Evaluation of Educational Achievement (lEA) has recently . . 
terminated a co-operative international project involving six subject areas 
at three different levels of education in each of 25 countries. Pour of these 
countries were developing countries. The procedures for evaluation have been 
well tested over several decades. The quicker such evaluation and experimentation 
procedures can be implemented under the auspices of the national ministries of 
education, for the benefit of their own systems of education, the better. Just 
as one author has shown that there are many dangers in implanting in one country a 
curriculum ^eveloped in another, so there are dangers in taking Instrumentation 
for evaluation purposes from one country to another. It is the evaluators and 
experimenters within each country who must develop instruments relevant to 
the needs of their own projects in their own cultures. 

^' EVALUATION AND EXPERIMENTATION STRUCTURES 

The competencies required to undertake such evaluation work could be 
collected within one central unit (for example within the Ministry of Education) 
or relevant planning organisation, or within a university, or in an independent 
organisation. Which location within the education system is optimal will depend 
to a large extent on the existing structures and where the competencies are 
currently found. - - 

The evaluation^experimentatioh component would have two major functions: 

(a) The systematic monitoring of the 'efficiency' of the educational* 
8y«5tem, in particular (i) student leaming outcomes, measurable . 
changes in student behaviours at various points in the educational 
system; and (11) structural criteria (financial, demographic, 
growth and change, etc.) and societal impacts (particularly as 
regards student placement in the Job market, changes in produc- 
tivity levels, etc.). 

(b) Experimentation with and evaluation of methods of teaching, 
curricular materials, instructional processes, etc. to determine 
their 'workability' before implementation is undertaken. 



The monitoring of the ayatem aa a whole euid the evaluation of 
innovations before implenentation will both generate information which can 
be of great help in further dec is ion- making in the aystera. It is essential 
that links between the evaluators and the policy-makers be built eucid maintained. 
One of the critical tasks in the setting-up of such experimentation and 
evaluation components will be that of convincing senior officia^ls in the 
appropriate ministries of the need for such work. 

The structure of such experimentation-evaluation units, as mentioned 
above, will depend on existing atructures within the system. Appendix I to this 
paper presents a series of comments on the way in which researcn centres dealing 
with evaluation and experimentation have evolved, the sources of demand for 
research, and the different structures of such centres • 

The word research is used in Appendix I simply because it is the general 

term which has been used to denote, inter alia , the experimentation and 

evaluation aspects of educational aasessment, . Ihe structures of the existing 

research competencies and units clearly vary from country to country. There 

would seem to be little use in a donor agency helping isolated individuals 

unless the basic number of personnel were brought together in a small unit 

with the * appropriate* relationships with the country's Ministry of Education 

and Planning. The major options include: improving existing structures, 

creating new units with specialised functions and, in some cases, using foreign 

pe2*80Dnel for short-term periods. 

For any proposed project for funding in a developing country, the 

donor's appraisal should, as a matter of routine, investigate the needs, 

feasibility, resources available, and possibly existing structures for 

evaluation, and propose the type of evaluation and experimentation \init 

to be set up. 

The extent to which the structure is centralized will be an issue. 
In general If no unit exists our recommendation would be to have a centralized 
unit placed within the Mlnistiry of Education. Such a centralized unit would 
include structures concerned with curriculum development, teacher education, and 
•''all aspects of evaluation. We are aware of the arguments of certain researchers 
that the research unit should be independent of the policy-making structure, 
but our experience does not support this view. If the policy-makers wish to 
take evaluation seriously they will; if not, not. 'Ihe important criteria are 
that the unit should have effective comBunlcatlon with the various levels of > 
policy-»akers, the ability to collect the required data from the schools and 
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students, and the ability and wherewithal to process and analyse the data. 
This in turn presupposes that there will be adequate financing for a small 
nucleus of staff and good physical facilities. Donor agencies should support 
the structure in such a way that it would survive long after the donor agency 
had finished its work ther^. This survival will depend to some extent on 
the recruitment undertaken, the way in which the career pattern fits into 
the structure, the intrinsic interest in the work, -etc. 

The relationship of the evaluation axid experimentation unit to the 
curriculum development centre, the examinations unit, and teacher training 
will be of prime importance to the functioning of the education system. All 
that can be done at this point is to raise the issues, 

THE TRAINIMO OP BVALUATORS 

n^e main aim of this section is to outline various approaches to the 
training of eTaluaton in deTelopinc oountries - an activity which is, as yet, 
weak in these countries. The training of evaluators must be recognlaed, both 
by donor agencies and the recipient country, as one of the priority areas for 
technical assistance. In the majority of countries, there is at least one 
person with knowledge of, and engagement in, evaluation. Such personnel who 
can participate in training should be identified, their own additional training 
needs should be specified, and the approaches to training planned. 
2.1 General situation 

There is In general a lack of trained evaluators ' in developing countrilss, 
but the situation varies from countXT- to country.. If one takes developing 
countries as a whole, it would appear that the actual experience in, and 
knowledge of, evaluation (in all its many aspects) is limited. There are 
certain characteristics of evaluation personnel in which different countries 
vary. The more Important characteristics are: 

(i-) The number of persons involved In evaluation work; 
(ii) The qualifications and the amount of experience of the persons 
involved in evaluation work. 
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(ill) Placement of educational evalua tors. In general peraona with 

evaluation competence tend to be widely scattered in developing 
countries. This soattering is associated with a lack of systematic 
and co-ordinated evaluation work, 
(iv) The nature of the experimentation and evaluation activities In 
which evaluation personnel are engaged. 
All these different characteristics can be identified for each country * 
and will help detenaine the type of training prograones required. 
2.2 possible modes of training > 

The needs of persona to be trained will differ somewhat from country 
to country and from time to time. One should perhaps think of training at 
two levels: simple and complex. Certain routine aspects of evaluation (e.g. the 
evaluation of the first and second try-outs of new curriculum materials) 
are elementary. Other aspects require a great deal of knowledge in research 
design, sampling, statistical analysis, etc., on the part of the evaluation 
leaders, in eaoh country. Unfortunately, those with more advanced knowledge 
but with little experience sometimes 'use a sledgehaj«er to craok a nut' . 
This tendency must be avoided. 

Let us consider the following training models: 

1. On-the-job training; 

2. Tailor-made international intensive training courses; 
1>. Regional and national training courses; 

4. Short-term academic training; 

5. Studir leaves. 

t- ' • ' ■ 

2-21 Qn-the-.lob training. One of the quickest ways of improving the. 

evaluation competency of a group is to require that they accomplish certain 
tasks within a certain time limit. But the tasks must be highly specific 
(e.g. construct a test to measure the science achievement of 8th graders, or 
draw a probability sample of- pupils aged 10 Ui the publl<^.aystem of education). 
An example ol this model has occurred in the course of the lEA co-operative 
form of research, which Increased- the overall competency considerably In many 
national research centres in this way. Each lEA national researcher had to 
go through a series of steps: identification of target populations, content 
analyses of textbook^, item writing, pre- testing of Items on specially selected 
Judgement samples^ item analysis, drawing of probability samples of schools 
and students in target populations. Identification and measuring of critical 
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social variables, translation and printing of all tests* briefing of tiastera,^ 
data collection* coding* data recording*, interpretation of analyses* etc. ^ 

Each national technical officer and his assistants had thus to calrry 
out in detail each step in the evaluation process for the national system 
in his own coufttry. 

Another form of on-the-job training is to. have a specific project 
begun* and for the donor agency to supply the appropriate technical assistance 
either for the period of the praject or at regular Intervals throughout the 
project. The technical assistants would have to be hand-picked. The selection 
fliechanism might pose a problem^ What is important in this type of training is 
that the group of evaluators remains the same from the conceptualisation of 
the project through the interpretation of the results and write-up. \One 
difficulty of this type of training is that for more complex data theVpmput^r 
facilities and/or programming personnel may not be available in' develc^ing 
countries whereas in the lEA type of activity the data processing would be 
undertaken internationally. In each case the bulk of the financing should 
come from the individual nation. 

2.22 Tailor-made international intensive courses . Intensive courses 

at ^-'international centre can be useful provided that the course is geared 
closely to the needs of the participants. In several instances* persons 
with virtually no knowledge or experience of ^evaluation have attendeST 
international courses and upon return to their countries have initiated 
evaluation activities which within two or three years became the mechanism 
for undertaking major work in education system pionitoring and assessment. 
This type of developoient can readily be brought about in developing countries. 
In many ways it should be easier when there is less tradition to be overcome. 

An example of the rationale and timetable of an intensive course given 
by the International Institute for Educational Planning (IIEF) is presented 
in Appendix II. Universities and testing agencies also provide training 
courses in evalxiaticn. The* total cost of such courses can run to $90*000 
(55 persons are all costs are borne by the sponsoring agency. 
2.25 Regional and national training courses . Again* these courses 

should be tailor-made to meet the needs of the participants. The advantage 
of regional courses is that they will cost less* and the evaluators in a 
region will get to know each other's work and can set up a collaborative 
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network. Such oouraea oould be of 4-6 week's duration with aa many faculty 
nefflbera aa poaalble being drawn fronj the region Itaelf, 
2.24 Short- and long^terw aoademlc training 

For ahort-tenn training. It might be beneficial In aeveral countries 
to Identify peraona who will become leadara in evaluation In their owA 
countrlea and offer them training. Such personnel ahould be selected In 
terma of their Interest (e.g. teacher effectlveneaa evaluation, curriculum 
evaluation, auirvey evaluation, advanced atatlstlcal analysis, etc.) for 
the formation of groupa of some 10-25 persons from varloua countrlea. Each 
group would have one or two Intensive couraes of three or six months' duration 
at one or two selected academic Inatltutlons, 

For long-term training. It has been oonmon practice In past decades 
for, evaluation apeclallata to attend universities In other countries. They 
attend for a three- or four-year period for their graduate work, there being 
no poaslblllty of graduate work In their own country. In many ways It would 
be of more practical value to have atudents train at a foreign university for 
abort perloda Interspersed by longer periods of field work In. their own 
countries. 

To this end It Is anticipated that one or two unlversltlea will be 
selected which can offer programmes' of three to four months* duration each 
year for a pe^rlod of three to four years. Although these programmes could 
lead to advanced degreea, the main thrust of the work would be found In 
the country, df origin. 
2.25 Study leave abroad 

For certain aenlor evaluators it would not be necessary to have further 
aoademlc training, but it would be desirable for them to gain 'knowledge on 
how the problems they are facing are dealt with in practice in other countries. 
It is suggested that individuals would have study leave of a few weehe' -duration 
to visit several evaluation centrea known to have made progresa in the particula 
areas of Immediate Impor^tance to the centre concerned in order to observe the 
methods of evaluatioiK elsewhere. 
2»3 L'eyels of expertise and existing resources 

Many persona make a distinction between simple and complex types of 
evaluation. Perhaps the simple type should be learned by on-the-job training 
and national regional aeminars, the more complex being reserved for other forms 
of training, Die more complex training would hav^ to be more aubject-matter 



anchored; e.g. teaching ef fectiveneae research would have to be a specialised 

oouraa. It Is difficult to see the borderline between the two clearly at 

this point, but this should become clearer as experience with training proceeds. 

3. EVALUATION COSTS OP TYPICAL PROJECTS 

No reference has yet been made to the coats of canning out 
any one evaluation of a particular project. Clearly the costs 
will vary from project to project, depending on local conditions and the size 
and complexity of the project, Below we have described three different types 
of evaluation of three different projects each of which might be regarded as 
typical, we show the timetable, the competencies required and the personnel 
man- months. 

Project 1. Summative evaluation 

Let us assume that a susBjative evaluation is required of a specific 
vocational training programme which is operating in 100 schools in a <deve loping 
country. The curriculum would be well established, but the test constructors 
would have had little experience of the vocational subject curriculum and no 
experience of developing tests in. that pubject. Testing would be carried out 
in some 40-50 schools with about 2,000 students in the sample involving data 
colleotlon on input prbcess wd output data. This would yield a considerable 
voluie of data. Hie analyses will Include multivariate! analyses, and hence 
it will not be feasible to have the data analysed by Hand. It is recommended 
that a coniputer be used, and this Implies periodio access to a computer system. 
(As a general point. It should be stated 'that -small computer facilities already 
exist in developing countries, and often they are iinderr-utillzed. The critical 
consideration is that the hardware be powerful .-enough to allow progranoing in 
a high level language such as FORTRAN or ALGOL. This kind of^ facility enables 
evaluation units to develop their own computer programme's,, and to take advantage 
of the, many packages created to do this sort of work in various 0Mtr«a arbusul 
the world.) 



(kie nenber of the evaluation unit will zct aa project director. 
The timetable will be ae follows: 



Construction of tests and Instruments (incliidlng 
' pre- testing) 

Drawing of saaple 

Printing of tests 

Adnlnlstratlon of tests 

Coding and recording of data 

Statletlca-1 analysis - — 

Intepretatlon of results and write-up 



Time In 
months 



Month 0 



Month 6 



Month 7 

— Months--8^9" 
Months 10-12 



The competehclps required will bej 

(a) test euid scale cpnst;cuctlon 

(b) sampling theory application 

(c) statistical analysis 

(d) data processing 



The persons required together with the tine In man-months (jbb) will be 

Professional ^ ^ " , 

(•|) 1 test constructor (2 mn) ^ 

(b) i curriculum consultant 2 m) 

(c) 1 scale and questionnaire constructor (2 mm) 

(d) 8 test administrators (2 mm) 

(e) 1 data pro<^es6(j|r (8 mm) 

(f) 1 statistician'^1 mm) . . ^ 
Total: 17 man-months 

(g) 2 coders (l tm)' ^ i ^ 

(h) 1 secretary (12 mn) 
Total: 13 ^am-months 
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^ 3.2 Pro.lect 2. Formative evaluation of curriculum material 

The plan of formative evaluation la baaed on the aasumptlon that a 
curriculum team haa prepared a di'aft of new Instructional material. This 
draft has been 2>rodaced in sufficient number of copies and tried out In a 
sample of approximately eight classes selected on Judgemental basis in such 
a Way that it represents the student population for which the progiramme has 
been prepared. 

The responsibility of the evaluators will be to collect data during ^ 
the try out of the material, to summarize the data, and to formulate suggestions 
if needed, for the modification of the programme. „___ 

The 'Wmetay,j9_-Of jsuch-ac-tivl^^ prescribed for 

the try out of the material. It is assumed that a course will be taught during 
a. full school y«ar, i,e,, during ten months. Evaluation activities should start 
two morithci before the actual try out starts, continue during ten months of the 
teachini^ period, and then continue for two additional- months after finishing 
the try out. 

The plan is prepared for a course which is taught during a whole^ school 
year to an extent of five periods in a week. Such a course would correspond ' 
^ to a one-year course in one of the major subjects taught in school, such as 
mother tongue, science, or practical agriculture. 

The timetable will be as follows: 



Contacting schools 

Construction of 7 fonnative evaluation 
instrument!^ (approximately 7x40 items) 

Visiting 3 .lools 

Administration of tests 

Interpretation of results 

The competencies required will be: 
r - (a) test construction^ 

(b) experience in teaching 

*■■ 
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Time in 
months 

Mon-pi 0 
Month 2 
Month 5 

Month 10 
Months 11-12 



- 11 



Th0 persons required with the tl»e In wm-^Months (mi) will be: 
Profegalonal 

(a) 1 test constructor (7 sub) 

(b) 1 teacher (2 ran) 

(c) 1 student In teacher training Institute or University 
Department of Psychology (5 mm) 
fotal: 14 m«ui-months 




(d) Clerical assistance (l6 mn) 
Total: l6 man-months' 

3.3 Project 3 . Formative evaluation of teacher education programme (s ) 

The purposes of the proposed formative evaluation of a teacher education 
programme would be to determine: (a) the objectives of the progr^aanie, (b) their 
appropriateness In the light of current knowledge concerning teacher oducatlon 
and teacher effectiveness, (c) the degree to which the objectives are being 
achieved, (d) the degree to which specific other objectives are desirable and 
feasible, axui (e) the estimated cost of installing a prograirane that would 
achieve more desirable objectives. Finally the project would eventuate in 
(f) reconmendations. 

Hie methods that would be used to achieve j-each of the aforementioned 
purposes are^ indicated below along with estimated of the competencies and 
man-months needed. 
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TIM In 



Examine literature oourae syllabi^ textbooks j 
manuals^ etc. of each of the oourses and other 
experienoes to be undergone by teaoher trainees 

Compare the materials^ Instructional and training 
methods^ and experienoes umdergone by the 
trainees with those considered desirable In the 
light of current knowledge 

Co nstructs ad alnlsj^rj *M in-hflrr^^-^qM*»^t,'^»"-^ 



nalre^ Interview^ euid observation Instruments 
for the sl^dents at various points In the 
teaoher education programme (earljj middle^ 
and late stages) 



In interviews and discussions with 
administrators^ polloy-makora^ teachers^ 
students^ altomi^ and employers of the 
products of the teacher education 
px*ogramne 

Examine the kinds of personnel^ material^ 
experiences^ equipment^ etc.j needed to 
carry out alternative prograflmie(8) 

Combine the results of Tasks (a) (e) 



I 

i' 

week 1 



week 1 1/2 



week 4 l/2 



5 1/2 



week 6 



The competencies required will be: 

(a) eijcperience in the theoi*y, research / * 
literature^ and practical jdiases 

of teacher education. 

(b) evaluation 

T^e persons required with the time ^ in man-weeks (mw) will be: 
Professional 

(a) specialist. in teacher education planning 
and programming (6 mw) 

(b) 2 graduate- student assistants In evaluation (8 1/2 mw) 
Total: 14 1/2 man-weeks ^ 

Other 

(c) clerical assistance (12 mw) 
Total : 12 man-weeks 

The extent to iriiloh iivtouilt evaluation needs to be encouraged will vary ^ 
somewhat from country to country. Ihe support required for the acoompanying 
atruotures will also vary, nie need for training is great but the' modes of increasing 
compotence in evaluations are manlfpld. 

tWhat are the priori tleii which donor agencies give to these problems and what ' 
aeohanlsms oan they employ to exieourage and build up such evaluation? 
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APPE>©IX I 

COMMEOTS ON THE ESTABLISHME^f^ OF EXPERIMENTAL- FYALUATION UNITS 

Although the following describes the establishment to date of 
research centres, 1% is hoped that donor agencicis might be able to cut 
many comers ahd establish a central co-ordinating and/or executing 
centre rapidly. 

Yates (The Role of Educational 'Research in Educational Change, 
Pacific Books, 1971) suggested that a typical development of educational 
research organisations seems tc have included the following : 

(a) the enterprise was founded by individuals, who for the 
most part worked within the universities with little or no 
financial support; 

(b) the universities then granted some recognition, often sparse 
and grudging; . . ' 

(c) after some demonstrated success, governments and government 
departments recognized the potential value of educational 
research and began ^to allocate public funds to support 
projects, proposed to be carried out by individuals or 
teams within universities; 

(d) at that point, the university d^ partments either grew into 
large educational research units or separate research institute* 
were formed. 

There are some notable exceptions where the universities have played 
a relatively minor role (e.g. the Federal Republic of Germany and the USSR) 
but in general the pattern for developing countries has been that the 
research Institutes evolved in the way described by Yates. 

The question arises as to how to institute educational research 
quickly in developing countries with relatively under-developed .educa tic. al 
research activities within the universities. The need is^elt to develop 
school systems and the accompanying evaluation as rapidly as possible, 
capitalizing on the experience of developed countries, whereby ten years' 
progress^can be made in perhaps two years for the particular developing 
country. Whether or not this is a viable proposition is unsure. In many 
cases there is no tradition of data collection: there are extremely few 
trained and qualified research staff: where the research, staff is trained 
It quite often consists of young persons who have been abroad and have not 
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yet the necessary educational status within their own countries to be able 
to lead educational research. As mentioned alx)ve, the older educational 
statesmen often do not understand educational research euid it is often ^ 
difficult for such groups to work harmoniously together. In some cases, there 
are blatant examples of nepotism, whereby unqualified persons are put Into 
senior positions. , There are problems vis-k-vis the existing university staff 
in terras of status: should the educational researchers have equal comparable 
status or should they have a higher status since the research required by 
the government needs the best possible people' trained in the subject in 
the country. 

Some institutions have initially undertaken the collection of educa- 
tional census data or some very /small pilot projects, or both. It would 
appear that those institutions v^ich have proved to be successful have 
from the very inception had certain delimited and specified goals to 
achieve, vrtiereas those which have been relatively xuisuccessful have often 
been create^ for the purpose of educational research without there being 
arty immediate specified goals for them to aim at. 
a) Source of demand if or research 

Educational research has been initiated by governments, by universitiesj 
by private centers and foundations and by individuals. 

In some cases, the government will establish an institution or center ^ 
to carry out the education research. In this case, the educational center 
is financed completely by the government and certain statutory controls are 
employed. In some cases, the center for research might be within the 
ministry of education or in another governmental department where other 
social sciences related to edupa.tlon may be luidertaking more educational 
work than other domains (e.g. sociology, economics, demography, etc. ) . 
The center may be located within a university, ijsually, but not always, 
where the. .university is a. State one. . 

In some cases, the ministry of education or other governmental depart- 
ments will create a ^bureau' or unit within the ministry or department, in 
ordv2r to distribute funds for research projects to universities and research 
institutions v^ether they be State or private. In some countries, the 
bureaux have an elaborate system for identifying the priorities to be given 
to particular aspects of the research. In Sweden, for example, the bureau 
within the National Board of Education, asks the other departments within 
the National Board of Education for their suggestions for research priori- 
ties for the next five years. Once these priorities have been put into some 



sort of systematic form, they are made as suggestions to the unlversltte?. 
and other research Institutes In Sweden, asking for comments on th^se priorities, 
as well as suggestions for other types of research and how these would fit 
into the present priority structure. Again the bureau will ask certain 
individuals^ knovm for their wisdom and perception In education, . to give 
their suggestions for priorities for research over a five, or In some cat'es 
a ten-year period. All of these suggestions are then sifted by the senior 
members of the bureau and finally brought into a priority list which then 
goe3 before the senior members of the whole of the National Board of Education. 
The National Board of Education at this point can make suggestions for the 
financial distribution for the coming year, and at the same time, make modlfi- 
cations for the estimated budgets for the coming years and for the amount 
of money for the bureau ^s expenditures. 

This, of course, is a very rational approach and in other cases the 
priorities for research are often left to one or two members of a bureau 

y 

within a ministry, with very few controls upon them, except that they have 
a fixed lDu<3get within which to work. In yet other cases, a bureau within a 
department may consist of only one or two persons, .arid In certain countries 
there is no fixed budget for research, and the bureau has to negotiate the 
amount for each research project, or oven each item of expenditure within 
a research project. 

It is perhaps worth pointing out at' this time that th^r^ Is a 
tradition In certain countries not to have budgets too well specified, the 
suggestion being that when a budget is specified, this will Jeopardize the 
possibility of having increased -funds at a later time. This may appear to 
be an irrational procedure but it does have certain advantage within 
certain cultural or power structures, ^jihere the ability to obtain funds- 
depends on the personality and connexions of an individual, he often feels 
that his obtaining of funds should be relatively secret and not open to 
attack from his competitors. 

' s 

Certain gremt-giving agencies arid bureaux have begun the idea of 
sequenced budgeting, i.e. step one is clearly defined, step two is less well 
defined and step three even lesn well defined. However, ^ 1t la recognized 
that the details' of step two will depend upon step one and t.hr»refore an 
adjusted budget for step two with a more specific plan will be submitted at 
the end of step one. However, an initial allocation is made for the total 
amount of money for all three steps (which is, of course, only a 'global 
estimate), (cf. American Educational Research Journal, Vol. 7, No. 3, May 1970). 
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Government creates outside agencies 

It is possible .for a goverjninent or ministry to establish an outside 
agency (i.e., not within the government) which has the responsibility for 
deciding priorities ;of research and educational innovation in general, 
and the distribution of funds for those projects. The body itself rarely 
carries out research. An example of this would be the Schools' Council 
iv England and Wales. In some cases,./a government department will Independent- 
ly undertake research on particular problems that concern the school system. 
For example, a department of labour (Canada) has undertaken a long study 
of secondary school students and their choices ^or employment. A planning 
department in another country (France) has undertaken a very large scale 
researcV* project that has had ao its object the:; decisions on th.e implemen- 
tation of the university structure in the country and the distribution of 
university students, over a long period of time. These research projects 
are seldom hqAOedlin close corin^xlon with departments of. education and 
hav^ in some oli'.cumstances Interfered with educational research institution pro 
grammes. In 'general they have utilized larger, scale resources than are 
available tp the educational research institutions that might have been 
connected with the on-going educational researcu in the country? The general 
problems of co-ordination in such cases have, as far as can be seen, never 
been faced. 

Universities and similar institutions having research units 
It is quite common for university departments or faculties (e.g., 
education departments and faculties of social science) to have their own 
research programmes which the members of the faculty i^ally decide conjointly, 
and then seek finance frorh either the university itself or from outside 
sources, e.g. > grant-giving agencies of one kind or another. As will be 
mentioned later, one of the big difficulties of many independent researches 
is to avoid duplication. 

In some cases, an Institute of research will be loosely attached 
to the university (e.g., the Ethno-sociological Institute in the Ivory 
Coast and the University of Abidjan). In this case the institute is usually 
supported from outside sources with perhaps the university covering only the 
overheads. 

21 . . 
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d) 



Private research centres and foundations 



Certain research foundations have been created which are independent 
bodies, i.e., independent of the government, although they will have govern- 
ment representation on their governing bodies. Such institutions as the 
Australian Council for Educational Research (ACER), the National Foundation 
for Educational Research (NFER), Scottish Council for Research in Education 
(SCRE), and the New Zealand Council for Educational Research (NZCER), are 
examples of this. Their money comes mainly from local educational authori- 
ties, govemmeht grants ajid grant-giving agencies. In most cases, they 
conduct the research themselves but In a few cases they will commission 
research from an individual or small group of persons.. This, however, is 
becoming rare. 
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APPENDIX II 

INTENSIVE TRAINING COURSE ON EVAmATION IN EDUCATION, IIEP, 

7 January - 1 February 1974 ^ 

The content for the course was determined by asking the selected 
candidates what fields of interest were most Important to them in 
their current and future work. Prom the returned questionnaires, 
it emerged that the predominant interests were In^the areas of systems 
«valua"^ion and the evaluation of learning materials and procedures. 
However^, some participants expressed an interest In guidance and selection 
procedures and, 'to a lesser extent, in the evaluation of school personnel. 
The detailed timetable given below reflects these emphasists. 

The course is comprise^d of the teaching of strategies of evaluation 
in all four areas of interest and the presentation, of the techniques 
by which such evaluation is accomplished. In the area of techniques, 
topics covered will incl de research design, the theory and practice of 
sampling, instrument construction, mei^ods of data collection, data 
processing, statistical analysis and interpretation and report writing. 

The learning situations Include lectures, fiiscusslon groups, practical- 
work, individual consultation witli faculty and individual project work. ; 
In assisting each participant in the conduct of his current of future wdrk, 
the individual project work has a special place aind provision for it is 
extended throughout the course* 
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The course fall* naturally into two parts. Thr first two weeks .^ro 
devoted to an introduction of the four topics and thc<^.bep.inninp of individual 
Atsignnents. Also durinp the first two weeks basic ideas in mental measure- 
went, statistical analysis and research desip,n are to be dealt with. The 
second two weeks continue the presentation of topics and tochniqtirs with 
increased emphasis on system and curriculiAn evaluation and upon the comple- 
tion of individual project work, Ouestionnaire and attitude scale con-stmc- 
tion are given special attention in these latter two weeks. 

Further to this, each participant was renuestrd to «;elect one problem/ 
project with which he would be concerned in thr near future and describe this 
in detail under the followinp headings: 

1. Aim(s) of project (giving also social educational context). 

2. Evaluation design. 

3. Sampling procedures. 

A, Instruments (questionnaires « tests etc.), 

5. Data collection 

6. Data analysis. 

7. Proposed write-up. 

8. Problems you face in the work. 

9. Facilities at your disposal in general, but particularly for data 
collection and processing. 

A major part of the individual work at the traininp: course was 
concerned with helping to solve the types of evaluation problems with which 
each participant would be faced on his returnfto his country. Indeed, each 
participant had co write an. end-of-coursc pancr dealing with the solution of 
hi» problems. 
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TimotaMr 

P ay 1 Session 1 9.15 

Srssion 2 11.13 

Srssion ^ K . 10 

Session 4 16. 3n 

2 Session 1 ) 
Session 2 ) 

Session 3 



Session 

Session 5 18.00 
Pay 3 'session 1 
Session 2 



Session 3 
Se'ssion 4 
Hay A Session 1 



Session 2 



Session 3 



- 10. AS Welconr hv Oirrctor. 

Administrativr Announcements. 
Introduction of F.icultv and P'-^rtir ipant*? . 

12.45 Description ^nH Fxplanation of rrorrrimme . 

- Ifi.On Evaluation for what? 

- 18. on Discussion . 

Croup work . to discuss inrM'vidual nroifct 
work and nrefcrred readinr' s s i rnnent . 

tasks of identifyinr and oncrat i ^a 1^ s i np 
objectives in four main themes: 

National systemfi 
Curriculum 

Selection and (Guidance 
School Frrsonne 1 

Pr.TCtica) work 'on opera 1 1 on a 1 i f i 
oh ject ivcs . 

1^.3^ Individual consultation. 

Individual consultation or lihr.irv worl' . 

"Introduction to the measurement of abilitv 
and achievrrent . ' This will include the 
concepts of scales of measurement, relia- 
bility, error, validity, di sners ion nnd 
covariation. 

'Practi cal e^erci ae. 

"Univariate statiFticn" nlus disctission. 

Fvaluation of pedaef^^i^al aspects of 
national systems. This will include the 
notions of efficiency and effectiveness 
and will deal with wastar.e (i.e. repetition 
and drop-out), sr lec ti'on "of indices and 
outcome criteria. 

Curriculum evaluation. This will include 
the identification of different levels of 
curriculum and nrovedure^ and the trv-out 
procedures for a Riven leai'ninr unit. 

^ Croup work Croup 1 Systems 

Group 2^ - Curriculum 
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"^'y 5 Sessibrt 1 . 

• ' Session 2 

Session 3 ) 
' ^ • .Session 4 ) 

Hay 6 ' U^m. 

P^y 7 Se8>ipn 1 

Sess'ion 2. 

Session 3 ) 
Session 4' ) 

Dajrjt Session r 
Session 2 
Session 3 
Session 4 

D_ay Q Sessibn 1 

Session 2 

Session 3 
Session 4 

p ay 10 Session 1 
Session 2 
Session 3 
Session 4 

Pay 11 Session 1 

Session 2 

Setsion .3 ) 

jSessio^n 4 ) 
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Correlation Analysi(«. This will include 
rroduct-momrnt. rank-nrd«r. point' bi-sprial 
biaerial. tetrachoric ;,nd phi (with reference 

Norm and Criterion referenced measures. 
Test construction (includinp item analysis)'. 
Practical exercises and discussion. 

Individual consultation. 
Selection and Guidance 
niscussion. 

-^Individual consultation and work. 

Observation and Rating Techninues. 
Survey Desipn. 
Experimental Design." 
Discussion. 

The conceptualisation and execution of some 
evaluation projects in developing countries. 

Selection of research design for Particular 
contexts. Selected proposal, hv participants. 

Continuation of Session 2. - 

Individual work . : 

Analysip of variance 

Introduction to multivariate analysis. 
Multiple rejtression. 
Practical exercises. 

Some non-parametric statistics and their 
applications. 

Practical exercises. / — ~- 

cu'rr'ent'oJ°f ^"'^^^'^^''^ Participants on their 
current or future proposal , for research, 
followed by discussion. 



Day lA 



Pa y 12 SrfSBion 1 
Seasion 2 
Session 3 

y 

Session 4 
Day 13 Session 1 

Session 2 

Session 3 
Session 4 
Session 1 
Session 2 
Session 3 

Session 4 

D ay 15 Session 1 ) 
Session 2 ) 

Session 3 

Session 4 

Pa y 16 Session 1 

Session 2 
Session 3 
Session A 
Pay 17 Session 1 

Session 

Session 3 
• Session A 



MoHols of currj^^tfunn evaluAtinn. 
IndividuarjS^ff\nT^(tion am 

Content anaf^i« , >(^printin^ |a>id itAfi writipp,. 
Practical exercise on Jtem writiiiPt. 

Sampling of"VarRet popul^tlrons (.curvrv .ind 
experimental). 

Selected presentations on sanplinp. by 
participants. 

Hnits of item analysis. 

Practical exrrci?»r - irom analysis. 

Oiiestionnaire dosip.n. 

Practical exorcise on questionnaire item vritinr. 

The nature of attitudes and other affect ivr 
constructs. 

Ptscusflion. ' ^ 

Attitude scale construction, scaling techniques 
and analysis. 

PractlLcal exercise. 

Individual consult-ntion. 

Need flnd.TTiethodolorv of rv^^lunrirn o<- technolopv 
of learning materi.qls . 



niscussion 
Individaal consul tntion. • 
Individual work. 

Introduction to data. Drocossinn - from rrccntion 
of data to final analysis. Simple approaches. 

Introduction to data procpssinr - from reccnt-lon 
of data to final analvsis. Sophi-JticatPd 
approach. 

A casp study of curriculum evaluation. 
Discunsion. 
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Sl^8f^ion ? 



Da^ 19 



\ 



Day 20 



Day 21 



Se«5f»ion 3 ) 

Session A ) 

Soasion 1 

Sesaion 2 |' 



) 
) 

) 
) 
> 
) 

Session 1 ) 
Session 2 ) 
Session 3 ) 

Session A ) 



Session 3 
Sesaipn A 

Session 1 
Session 2 
Session 3 
Sessfon A 



Multiple repression and canonical analv«;iQ 
followed by discussion. 

Individual work nnd consultation. 



Itefh sampling. - i^otfltion and Rasch. 
Discussion. 



Individual vork and consultation. 



Participants,' presentation of complptrri project 
designs plus discussion. 



Participants' presentation of completed project 
designs plus discussion. 

Director's concludinr renarks 
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